Skip to content

fix(server): drop unexpected PDUs during deactivation-reactivation#630

Merged
Benoît Cortier (CBenoit) merged 12 commits intoDevolutions:masterfrom
elmarco:react-fix
Jan 27, 2025
Merged

fix(server): drop unexpected PDUs during deactivation-reactivation#630
Benoît Cortier (CBenoit) merged 12 commits intoDevolutions:masterfrom
elmarco:react-fix

Conversation

@elmarco
Copy link
Copy Markdown
Contributor

The current behaviour of handling unmatched PDUs in fn read_by_hint() isn't good enough. An unexpected PDUs may be received and fail to be decoded during Acceptor::step().

Change the code to simply drop unexpected PDUs (as opposed to attempting to replay the unmatched leftover, which isn't clearly needed) and make single_sequence_step_read() resilient to decode issues when ignore_unexpected_pdu. This require Acceptor::step() to be idempotent on error, so we have to clone the state fields, instead of taking the state.

Comment thread crates/ironrdp-async/src/framed.rs Outdated
Comment thread crates/ironrdp-async/src/framed.rs Outdated
Copy link
Copy Markdown
Member

@CBenoit Benoît Cortier (CBenoit) left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

Change the code to simply drop unexpected PDUs (as opposed to attempting to replay the unmatched leftover, which isn't clearly needed)

Sounds reasonable!

This require Acceptor::step() to be idempotent on error, so we have to clone the state fields, instead of taking the state.

To be honest, I’m not very inclined into making this change, as I believe there is a better approach.

Before going further, I want to check my understanding at this place: https://github.com/Devolutions/IronRDP/pull/630/files#r1905600306

@elmarco
Copy link
Copy Markdown
Contributor Author

I added a few other deact-react fixes to this PR

Comment thread crates/ironrdp-dvc/src/server.rs Outdated
@CBenoit
Copy link
Copy Markdown
Member

Benoît Cortier (CBenoit) commented Jan 8, 2025

I added a few other deact-react fixes to this PR

I’m not sure what deact-react means in this context (EDIT: deactivation-reactivation, got you 😂), but all of these new commits are looking good to be merged. 😄
I still need some time to think about the first one though.

Copy link
Copy Markdown
Member

@CBenoit Benoît Cortier (CBenoit) left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, sorry for the delay.

I gave a bit more of thought to this, and it’s indeed not so simple to do better as I initially thought.

Let me go over the problem we are solving, and reformulate my concerns and questions I have with the way it’s implemented now.


We are trying to be as resilient as possible by ignoring anything we don’t expect for the current accept sequence state. This means:

  • Things that are not matching the provided hint.
  • Things that can’t be decoded into the expected structure.

This happens because as I understand it, the client does not stop sending active session packets (e.g.: FastPathInput PDUs, etc) even though the deact-react sequence is already started.

Ideally, we would rely on some synchronization logic, a "fence", to agree with the client on when packets relevant for the reactivation (~= connection/accept) logic should be send and received.

In the absence of such synchronization logic, we need to run the sequence state machine, without consuming its state even if we failed at decoding the expected structure from the provided frame. This results into having to clone everything because of the way we are consuming the current state to turn it into the next one.


Here are some of my concerns:

  • It’s not very clean to clone our way through this. I acknowledge that this is a possible fix, and I would be okay to merge this as a stopgap solution if you think it’s the best we can do with minimal efforts.

  • Typically, servers are just killing the connection brutally when the client is behaving in some weird way. Microsoft’s official server is doing this. I feel we are doing something wrong if we need to accept what the official server would usually reject.

  • From the above: instead of terminating the connection as soon as we get something weird, we ignore the problem, and the session may stay hanging around, doing nothing useful for a while.


I investigated the RDP deact-react sequence more in depth, to better understand the domain and possible reasons that may be leading to this problem in the first place.

Normally, the deact-react sequence works as follows:

  • The server initiates the deactivation. When the server needs to tear down the current share or reconfigure it, by sending a Deactivate All PDU.

  • Upon reception of the Deactivate All PDU, the client must stop sending active-share graphics updates (both FastPath updates and SlowPath updates).

  • The server begins reactivation by sending a new Demand Active PDU, which carries the new share settings.

  • The client then responds with Confirm Active (and possibly other PDUs like Synchronize, Control, Font List, etc.).

The key point is that once the server has sent Deactivate All, the client is expected to stop sending further PDUs refering to the old context, and wait for the new handshake. In principle, it should be fine to just fail upon reception of an unexpected payload.


At this point, I suspect we may be hitting a race condition depending on the network latency with client’s send buffer hitting the wire even after the server decided to deactivate, and by the time these packets arrive, the server has already started the reactivation process. Or, there is a client-side bug. For instance, I’m not sure if the IronRDP client is properly adhering to the deactivation order sent by the server.


Best approach:

  • Fix IronRDP client if it does not properly follow the logic to quiesce old PDUs on Deactivate All.
  • Send the Deactivate All and Demande Active PDUs to the client, and wait for the Confirm Active PDU. Any PDU received in between should be discarded.

The key point to this approach is that we just need to parse received frames until a Confirm Active PDU is received, and then we can drive the sequence as usual.

In pseudo-code:

let frame = loop {
    let frame = read_frame_by_hint();
    
    match decode_confirm_active_pdu(frame) {
        Ok(_) => break frame,
        Err(DecodError) => {}
        Err(error) => return Err(error),
    }
};

// Feed the frame, which is just the PDU that is expected for the next step.
// We decode the frame twice, but from this point on we don’t need to check for the decode result again.
accept_sequence.step(frame);

// At this point, drive the sequence as usual.

=> We pump and discard all the messages exchanged between the Deactivate All and the Confirm Active PDUs.

Less ideal approach:

If we really need the ability of handling unexpected PDUs at any point in the accept sequence, then maybe a better approach is to try decoding the next expected PDU for the current state before actually executing the step. We may need to add an extra trait in the ironrdp-acceptor crate for this purpose.

Less ideal approach 2:

Move data out of the state variants, and use Options directly at the ClientConnector so that we can set and take the various states without cloning everything, and it’s possible to leave the state machine untouched until the expected PDU is received.

Extra remark:

Definitely something for a separate PR, but we may want a timeout during the connection and deact-react sequences, in case the client decides to do something akin to a “slow” or “lazy” DoS. An RDP server should never be exposed to the internet so this is debatable whether an attack could actually take place though.


Do you think the "best approach" (as I call it) would work?
Would you prefer that we still merge this PR as-is and improve on it afterwards?

I’m open to any of your suggestions at this point.

@elmarco
Copy link
Copy Markdown
Contributor Author

[...]

The key point is that once the server has sent Deactivate All, the client is expected to stop sending further PDUs refering to the old context, and wait for the new handshake. In principle, it should be fine to just fail upon reception of an unexpected payload.

At this point, I suspect we may be hitting a race condition depending on the network latency with client’s send buffer hitting the wire even after the server decided to deactivate, and by the time these packets arrive, the server has already started the reactivation process. Or, there is a client-side bug. For instance, I’m not sure if the IronRDP client is properly adhering to the deactivation order sent by the server.

I am also testing with mstsc, and freerdp.

Best approach:

* Fix IronRDP client if it does not properly follow the logic to quiesce old PDUs on Deactivate All.

I don't know if it's necessary to do anything.

* Send the Deactivate All and Demande Active PDUs to the client, and wait for the Confirm Active PDU. Any PDU received in between should be discarded.

The key point to this approach is that we just need to parse received frames until a Confirm Active PDU is received, and then we can drive the sequence as usual.

In pseudo-code:

let frame = loop {
    let frame = read_frame_by_hint();
    
    match pdu::decode(frame) {
        Ok(_) => break frame,
        Err(DecodError) => {}
        Err(error) => return Err(error),
    }
};

// Feed the frame, which is just the PDU that is expected for the next step.
// We decode the frame twice, but from this point on we don’t need to check for the decode result again.
accept_sequence.step(frame);

// At this point, drive the sequence as usual.

Sounds reasonable, I'll see if I can make it work.

@elmarco
Copy link
Copy Markdown
Contributor Author

not tested very carefuly yet, it would be worth to write a test for it.

Comment thread crates/ironrdp-acceptor/src/connection.rs Outdated
Comment thread crates/ironrdp-acceptor/src/finalization.rs Outdated
Comment thread crates/ironrdp-acceptor/src/finalization.rs Outdated
Comment thread crates/ironrdp-async/src/framed.rs Outdated
@CBenoit
Copy link
Copy Markdown
Member

Benoît Cortier (CBenoit) commented Jan 18, 2025

not tested very carefuly yet, it would be worth to write a test for it.

Understood! I’m fine with merging before the test is written.

My last concern is about the extended interface for the Sequence trait: #630 (comment)

If that’s necessary I’m happy to go that way, but I failed at understanding why it became necessary given that the previous version of this PR was not doing anything similar.

EDIT: I’m sorry, I broke the CI when I tried to add an in-line comment to keep a note of the undocumented behavior using GitHub UI… I can fix that once I have access to an actual git CLI later! Or feel free to squash into your commit =)

@elmarco
Copy link
Copy Markdown
Contributor Author

still working on some tests, do not merge :)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
For completeness, this error is used by FreeRDP.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
@CBenoit Benoît Cortier (CBenoit) force-pushed the react-fix branch 2 times, most recently from 54d3c4b to cd4e662 Compare January 27, 2025 12:14
The current behaviour of handling unmatched PDUs in fn read_by_hint()
isn't good enough. An unexpected PDUs may be received and fail to be
decoded during Acceptor::step().

Change the code to simply drop unexpected PDUs (as opposed to attempting
to replay the unmatched leftover, which isn't clearly needed)

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
I couldn't find any explicit behaviour described in the specification,
but apparently, we must just keep the channel state as they were during
reactivation. This fixes various state issues during client resize.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
This is just some basic tests, but hopefully it will grow to be more
friendly and cover more behaviours.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
This makes freerdp keep the flag up and handle desktop
resize/deactivation-reactivation. It should be okay to advertize,
if the server doesn't resize anyway, I guess.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
This makes code slightly nicer and allow further code reuse.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
It's problematic when the client didn't resize, as we send bitmap
updates that don't fit. The client will likely drop the connection.
Let's have a warning for this case in the server.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Copy link
Copy Markdown
Member

@CBenoit Benoît Cortier (CBenoit) left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!!

@CBenoit Benoît Cortier (CBenoit) merged commit 0f9877a into Devolutions:master Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants